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Abstract 

Background. In 2010, a new telephone service, NHS 111, was piloted to improve access to urgent 
care in England. A unique feature is the use of non-clinical call takers who triage calls with com- 
puterized decision support and have access to clinical advisors when necessary. 
Aim. To explore users' acceptability of NHS 111. 
Design. Cross-sectional postal survey. 
Setting. Four pilot sites in England. 
Method. A postal survey of recent users of NHS 111. 

Results. The response rate was 41% (1769/4265), with 49% offering written comments (872/1769). 
Sixty-five percent indicated the advice given had been very helpful and 28% quite helpful. The 
majority of respondents (86%) indicated that they fully complied with advice. Seventy-three per- 
cent was very satisfied and 19% quite satisfied with the service overall. Users were less satisfied 
with the relevance of questions asked, and the accuracy and appropriateness of advice given, 
than with other aspects of the service. Users who were autorouted to NHS 111 from services such 
as GP out-of-hours services were less satisfied than direct callers. 

Conclusion. In pilot services in the first year of operation, NHS 111 appeared to be acceptable to the 
majority of users. Acceptability could be improved by reassessing the necessity of triage questions used 
and auditing the accuracy and appropriateness of advice given. User acceptability should be viewed in 
the context of findings from the wider evaluation, which identified thatthe NHS 111 pilot services did not 
improve access to urgent care and indeed increased the use of emergency ambulance services. 
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Introduction 

In 2010, a new telephone service NHS 111 was launched in four 
pilot areas in England. The aims of the service were to improve 
access to and satisfaction with urgent care, make more efficient 
use of emergency and urgent care services by directing people to 
the most appropriate service, and in the longer term reduce use 
of emergency ambulances (1). Members of the general popula- 
tion called a free, easy to remember number, '111', and spoke 
to a trained non-clinical call taker who used the software NHS 
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Pathways to direct people to the most appropriate health care. 
Call takers had access to clinician support if needed, usually a 
nurse. The service also took telephone calls on behalf of gen- 
eral practice (GP) services operating outside the normal working 
hours of Monday to Friday during the day ('GP out-of-hours 
services'). Calls to GP out-of-hours services were automatically 
put through to NHS 111, or a message on GP answer phones 
directed people to call NHS 111 out-of-hours. The new service 
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was an addition to a set of services meeting emergency and 
urgent care needs in England including emergency ambulances, 
emergency departments, urgent care centres, GP out-of-hours 
services and daytime GP. The four pilots operated while NHS 
Direct, a 24-hour nurse-led telephone service for health advice, 
was part of the emergency and urgent care system, although the 
national policy plan was to replace the telephone service pro- 
vided by NHS Direct with NHS 111 during 2013. 

The NHS 111 pilot service was estabhshed in response to 
confusion in the general population about how to access services 
for urgent care (2,3). The rationale was that NHS 111, with an 
emphasis on offering people an easy and fast response to an 
urgent situation, could answer the telephone immediately and 
direct people to the 'right service, first time'. It is important to 
identify users' views of this service because the consequences of 
poor acceptability may be lack of adherence to advice or treat- 
ment, complaints and litigation or avoidance of future use. The 
aim of this study was to explore users' views of NHS 1 1 1 in four 
pilot sites prior to national roll out of the service to highlight 
any improvements needed for the national service, as well as 
provide evidence about future international models of telephone 
access to care using non-clinical call takers. 

Method 

We undertook a cross-sectional postal survey of recent users of 
NHS 111 in four pilot sites between July and October 2011, 
around 9-1 1 months after the start of the service. We planned 
to send questionnaires to 1200 users in each site. Calls made 
in a single week were used as the sampling frame for sites with 
a large number of calls. A 2-week sampling period was used 
for sites with lower numbers of calls. When >1200 calls were 
identified in a week, systematic random sampling was used to 
select 1200 calls. After sampling, a small number of calls were 
excluded by NHS 111 staff: patients aged <15 years if algo- 
rithms related to sexual health used because the questionnaire 
was addressed to the parent/guardian of this age group; users 
who had not provided their home address details and the second 
or more calls of repeat callers in that time period. 

Personnel at each site sent a covering letter, information 
booklet, questionnaire and reply-paid envelope to the patient 
within 3 weeks of the call. In most cases, the caller and the 
patient were the same person. Where calls were made on behalf 
of children, we addressed the questionnaire to 'care of the par- 
ent/guardian of. We asked in the covering letter that both caller 
and patient attempt to complete the questionnaire together if 
relevant and possible. Responses were returned directly to our 
team. Questionnaires had unique identifiers and sites were 
informed of which users needed reminders. Up to two remind- 
ers were sent to non-responders ~3 weeks and 6 weeks after the 
initial mailing. 



The questionnaire 

We asked how people had accessed the service and the advice they 
were given. Then we addressed the multidimensional aspects of 
satisfaction (4), asking about patient-centred care (helpfulness of 
staff, reassurance), access (clarity about when to use the service), 
communication and information (relevance of questions asked), 
technical quality (whether the advice worked well in practice) and 
efficiency (speed with which problem as dealt with). Most of the 
questions about satisfaction had response sets of 5-point Likert 
scales from 'strongly agree' to 'strongly disagree'. We included a 
question about overall satisfaction, on a 5-point Likert scale from 
'very satisfied' to 'very dissatisfied'. We also addressed two poten- 
tial consequences of poor acceptability: compliance with advice 
and views of whether they would return to the service with a simi- 
lar health problem. We measured caller demographics. Because 
these types of surveys have been criticized for ceiling effects (4), we 
added an open question to identify aspects of dissatisfaction as well 
as satisfaction (5). We gave respondents a short section to describe 
in their own words anything with which they were particularly 
satisfied or dissatisfied. The questionnaire was similar to one used 
in a previous evaluation of telephone delivered health care by NHS 
Direct (6). We used similar questions to allow comparison between 
the findings from the evaluations of pilots of NHS 111 and NHS 
Direct. We undertook a small pilot with three NHS 111 users in 
one site who discussed within a telephone interview the face and 
content validity of the questionnaire. We were aware before we 
embarked on this survey that NHS 111 users might not know that 
they had called the service because some users were autorouted 
from other services such as GP out of hours. We designed the cov- 
ering letter and questionnaire to address this. 

Analysis 

Data were analysed in PASW Statistics version 18. Not every- 
one answered all questions. Missing values were excluded, so we 
report the denominator for all results. We described the views of 
respondents, reporting 95% confidence intervals for key statis- 
tics. We undertook further analysis on the set of questions using 
the response set 'strongly agree' to 'strongly disagree, using the 
Friedman test to identify whether satisfaction differed by aspect of 
the service. Then we dichotomized the satisfaction items at 'strongly 
agree' versus other categories because people who use the extreme 
category of a satisfaction variable tend to have no negative com- 
ments to make, whereas people who use any other category can 
identify issues they were unhappy about (7). We undertook a logis- 
tic regression to test whether satisfaction differed by route into the 
service, adjusting for age group (IG-AA, 45-64, 65+), gender (male, 
female) and ethnicity (white, ethnic minority group). We analysed 
the open question using a 'quantitative strategy' similar to content 
analysis (8). One researcher (EK) read the comments and identified 
a thematic framework developed inductively from the comments. 
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EK then applied the coding frame to each comment, counting the 
numbers of respondents making different types of comments. 

Results 

The response rate was 41% (1769/4265), with 49% (872/1769) 
of respondents making written comments. The majority of call- 
ers were female, younger than 65 years and white (Table 1). The 
median age was 45 and interquartile range 32-63. 

Accessing NHS 111 

Two-thirds of respondents had dialled NHS 111 directly, with 
others 'autorouted' from another service or directed from 
another service (Table 2). Eighty-six percent (1495/1731, 95% 
confidence interval: 85% to 88%) reported being 'definitely 
clear' about when to call NHS 111. 

Advice given 

A fifth of respondents reported being transferred to the ambu- 
lance service or sent an ambulance, and a tenth reported advice 
to self-care only (Table 2). Respondents were asked how helpful 
they found the advice given by NHS 111. Overall, two-thirds of 
respondents reported receiving 'very helpful' advice from NHS 
111 (65% 1108/1695, 95% confidence interval: 63% to 68%). 



Table 1. Respondent demographics 



Characteristic 




nIN 


Female 


71 


1237/1750 


Age 






16-44 


47 


832/1754 


45-64 


28 


483/1754 


>65 


25 


439/1754 


White 


85 


1479/1736 


Disability or long-term illness 


34 


577/1718 


Home owners 


60 


969/1610 



Table 2. Reported route into NHS 111 and advice given by NHS 111 



Respondents were more likely to feel that the advice given was 
very helpful if this advice directed the patient to the ambulance 
service (76%) or if an appointment was arranged for the patient 
(71%) than if they were asked to self-care (64%), visit a health 
centre (55%), contact an 'other' service (54%) or contact their 
own GP (52%; P = 0.001). 

Satisfaction with different aspects of NHS 111 

Respondents were asked a closed question about their overall sat- 
isfacdon with NHS 111. Seventy-three percent (1255/1726, 95% 
confidence interval: 71% to 75%) were very satisfied with the way 
NHS 111 handled the whole process, 19% (319/1726) were fairly 
satisfied and 5% (79/1726) were dissatisfied. Eight hundred and 
seventy-two respondents provided written comments, of whom 
867 had also completed the closed satisfaction question. Seven 
percent (63/867) of people completing the open question reported 
that overall they were dissatisfied with NHS 111 compared with 
2% (16) of the 859 who did not complete the open question. 
That is, more people who were dissatisfied took the opportunity 
to write comments than those who were satisfied. 

Respondents were asked to 'strongly agree' through to 'strongly 
disagree' on a 5-point Likert scale with a series of positive state- 
ments about NHS 111 (Table 3). Small percentages of respond- 
ents disagreed or strongly disagreed with these statements. Some 
statements had different distributions of responses than others 
(P = 0.001). Two aspects of the service were less acceptable than 
others based on these closed questions and the responses to the 
open question (Table 3). There were concerns about the relevance 
of questions asked and whether the advice given worked in prac- 
tice. These two statements had the lowest proportion of respond- 
ents strongly agreeing. The themes developed inductively from the 
open comments mapped onto the aspects of the service addressed 
in the closed questions. More people chose to write positive com- 
ments about the service than negative comments except for the 
issue of relevance of questions asked. Indeed, 8% of all written 
comments were negative comments about the relevance of ques- 
tions asked, showing the relative dissatisfaction with this aspect 



% nIN 



Route mto NHS 111: dialled NHS 111 directly 


62 


1059/1712 


Called another service such as general practice out of hours and put through automatically to NHS 111 ('autorouted') 


24 


416/1712 


Called another service, usually their own general practice out of hours, and heard a message directing them to call NHS 111 


11 


193/1712 


Unsure 


3 


44/1712 


Reported advice given: NHS 111 arranged an appointment at a centre such as an urgent care centre 


34 


578/1690 


Transferred to the ambulance service or sent an ambulance 


20 


346/1690 


Told to visit a centre such as an emergency department, walk in centre or urgent care 


15 


256/1690 


Contact their own general practice 


9 


154/1690 


Self-care 


10 


167/1690 


Other (e.g. a health visitor, dentist, district nurse) 


11 


189/1690 
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Table 3. Satisfaction with different aspects of tlie NHS 111 service 



Statement 



Strongly agree Agree Neither Disagree 



Strongly N=100% 
disagree 



Positive comments''' 



Negative 
comments '■' 



The 111 staff 
'were helpful 



The questions 
asked by the 
111 service were 
relevant 



The 111 service 
dealt with my 
problem quickly 



The advice I was 
given by the 111 
service worked 
well in practice 



The 111 service 
helped me to 
make contact 
with the right 
health service 



63% 



30% 



4% 



1% 



1% 



1725 



50% 



36% 



5% 



2% 



1688 



58% 



31% 



6% 



3% 



2% 



1702 



50% 



35% 



9% 



3% 



2% 



1651 



53% 



32% 



9% 



3% 



2% 



1605 



N = 245 'They were 
very caring, friendly 
and reassuring' 



N = 38 'Satisfied: 
questions asked 
were relevant and in 
detail assuring me 
that I was talking to 
the right person' 

N = 172 'They 
didn't take ages 
to help me, they 
assessed what my 
problem was and 
got through the 
emergency questions 
quickly efficiently 
to get on with what 
was wrong' 



N= 118 'Ifeh very 
reassured that I was 
being given good 
advice and steps to 
follow' 



N = 100 'I had the 
help I needed in a 
matter of minutes, 
an ambulance came 
so very quick and a 
paramedic came in 
a car before that' 



N = 14 'Never 
mentioned if 
she gets worse 
ring all he said 
ok bye wait for 
their call - how 
rude' 

N = 78 'The 
checklist the 
operator read 
was irrelevant 
and she 
apologised for 
this' 

N = 27 'It took 
what seemed 
like a long time 
to discuss with 
first person 
on the line, to 
then be passed 
on to someone 
more qualified, 
to then have 
to discuss 
again when all 
I wanted was 
an appointment 
with the out of 
hours service' 
N = 59 'Was 
advised the 
walk in centre 
could deal with 
my issue, only 
to then be told 
by the walk 
in centre they 
couldn't help' 
N = 42 'After 
waiting 3 hours 
for a doctor to 
return my call 
we went back 
to the walk in 
centre where we 
were told 111 
had gotten the 
wrong phone 
number despite 
being given it 3 
times' 
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Table 3. Continued 


Statement 


Strongly agree 


Agree 


Neither 


Disagree 


Strongly 
disagree 


N = 100% 


Positive comments* 


Negative 
comments"' 


Using the 111 


55% 


30% 


9% 


4% 


3% 


1679 


N= 138 'The 


N = 3 'The 


service resssured 














service that 111 give 


service did not 


me 














IS exceptional. He 
[NHS 111 advisor] 
calmed me down 
and reassured me 
everything was 
going to be ok' 


reassure me 
and I was later 
admitted to 
hospital as an 
emergency' 


I was completely 


59% 


28% 


7% 


4% 


3% 


1706 


'Brilliant service, 


'I think it 


happy with the 














start to finish' 


[NHS 111] 


111 tiprvifp 
















IS ^ pomnlpfp 
waste ot i\rio 
resources and 
yet another 
hair brained 
scheme by the 
government' 


The 111 service 


65% 


24% 


6% 


2% 


3% 


1711 







is a valuable 
addition to the 
NHS 



of the service. (Table 3 ) . The next largest set of negative comments 
related to dissatisfaction with the advice given because people did 
not think that they had been directed to the right service. 

Compliance with advice and willingness to use 
service again 

The majority of respondents reported complying with all the 
advice given by NHS 111 (86%, 1435/ 1670, 95% confidence 
interval: 84% to 88%). A similar proportion reported being 
willing to use the service again for a similar health problem 
(86%, 1490/1729). 

One in 10 comphed with some of the advice (11%, 180/1670) 
and only a few people reported that they had not followed any of 
the advice given (3%, 55/1670). Respondents who did not fully 
comply with the advice given by NHS 111 (n = 235) were asked 
to indicate their main reason for not following the advice. Most 
respondents provided a reason for not complying (70%, 165/235), 
reporting that they did not agree with the advice (21%, 35/165), 
felt unable to follow it (20%), 30/165), felt that it did not work 
(19%, 32/165), felt that the health problem changed (7%, 11/165) 
or that they did not understand the advice (2%, 4/165). A third 
of people ticked 'other reason' that consisted of a diverse range of 
responses including 'I forgot' or 'it was too late' (30%, 50/165). 
While reported compliance was high, it differed by type of advice 
given. Respondents reported being more likely to fully comply with 



advice regarding the ambulance service (92%) or if an appoint- 
ment had been arranged for the patient (91%) than other types of 
advice: self care (85%), visit a health centre (83%), contact own 
GP (78%,), contact an 'other service' (74%; F = 0.001). 

Differences by route into NHS 111 

The NHS 111 call handling services in the four pilot sites were 
operated by different providers (one by an ambulance service 
and three by NHS Direct). There were differences in users' 
views between sites. We tested for differences taking age, gen- 
der and ethnicity of users into consideration because these can 
affect users' views. We also took the route into NHS 111 into 
consideration because some sites had higher proportions of 
direct users than others. We found that users' views differed by 
the route into NHS 111 (Table 4). Respondents reporting that 
they had been autorouted to NHS 111 from another health 
service such as GP out of hours were less satisfied than those 
who had dialled '111' or received a health service telephone 
message to dial '111', even after adjusting for demographic 
characteristics. Once demographics and route into NHS 111 
was adjusted for, there was only one statistically significant 
difference between sites. A higher proportion of users in two 
sites were clearer about when to use the service (90% and 
91% compared with 77% and 83% in other sites, adjusted 
F = 0.014). 
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Table 4. Percent of respondents 'strongly agreeing' with satisfaction statements by route into NHS 111 



Statement Direct dial % (n) Autorouted % (n) Message to redial % (n) Adjusted P value" 



The 111 staff were helpful 68 (713) 

The questions asked by the 111 service were relevant 53 (539) 

The 111 service dealt with my problem quickly 62 (640) 

The advice I was given by the 111 service worked well in practice 54 (541) 
The 111 service helped me to make contact with the right health 57 (557) 
service 

Using the 111 service reassured me 59 (598) 

I was completely happy with the 111 service 63 (649) 

The 111 service is a valuable addition to the NHS 70 (721) 

Overall satisfaction (v satisfied) 76 (791) 

N 1035 



^Adjusted for age group, sex and ethnicity. 

Discussion 

Summary and comparison with existing literature 

Overall satisfaction with NHS 111 was very good, with 73% 
of respondents reporting that they were very satisfied with the 
new service and 91% that they were very satisfied or satisfied. 
Caution is required when making comparisons with other ser- 
vices because satisfaction is often measured and reported in 
different ways. Given this limitation, satisfaction appeared to 
compare well with satisfaction with telephone consultations in 
GP settings: 98% (satisfied or very satisfied) (9), 88% (very or 
fairly satisfied) (10) and 62% (very satisfied) (11). It also com- 
pared well with international evidence on general population 
based telephone triage services at 89% (satisfaction score of 7 
or above out of 10) (12) and 73% (very satisfied) (13). One of 
our pilots had offered a similar service prior to operating NHS 
111, and evaluators of that service found very high levels of user 
satisfaction (14). 

The percentage of users finding the advice very helpful was 
65% compared with 76% of users of the NHS Direct pilots in 
their first year of operation (6), even though the age distribu- 
tion of NHS 111 survey respondents was older than that of 
NHS Direct survey respondents: median 45 (interquartile range 
32-63) versus median 35 (interquartile range 29-47) (6) and 
older people are usually more satisfied with health care (15). 
That is, early users of NHS 111 reported finding it less helpful 
than early users of NHS Direct. The differences in satisfaction 
and indeed the differences in age distribution between NHS 111 
and NHS Direct at pilot stage may be explained by differences 
in the source of users: NHS 111 took calls for GP out-of-hours 
services while NHS Direct did not do this at pilot stage. 

The majority of our respondents (86%) indicated that they 
complied with all of the advice given, comparing well with com- 
pliance rates for other telephone triage services which ranged 
from 56% to 98% (median 77%) (16). A surprising finding was 
that 86% of users reported being 'definitely clear' when to call 



53 (212) 


67 (126) 


0.000 


47 (184) 


52 (96) 


0.139 


49 (194) 


62 (114) 


0.001 


42 (159) 


52 (93) 


0.001 


45 (164) 


57 (102) 


0.001 


48 (187) 


56 (102) 


0.004 


50 (197) 


62 (117) 


0.001 


57 (229) 


65 (123) 


0.001 


64 (259) 


76 (144) 


0.001 


408 


189 





NHS 111 even though this was a new service and only 62% had 
actually dialled 111 directly. This may be related to the extensive 
use of local publicity about when to use the service in some of 
the sites (17). 

Two aspects of the service had lower levels of satisfaction — 
the relevance of questions asked and the advice given. Both of 
these concerns were also expressed by users of a similar service 
in one of the pilot sites (14). There was no evidence that the 
acceptability of the service differed by site. However, there was 
some evidence that people who call their GP out-of-hours ser- 
vice and expect to speak directly to them were less satisfied than 
people who chose to call NHS 111. There was no evidence that 
users were concerned about non-clinical call takers. 

Strengths and limitations 

This was a large survey of users of NHS 111 and provides the 
first evidence of satisfaction with this new service and with tel- 
ephone triage by non-clinical call takers. The response rate of 
41% was low. It is comparable with other large postal surveys 
of access to GP in England, which obtained response rates of 
41% m 2008 and 38% m 2009 (18), and a telephone survey 
of a service similar to NHS 111 with a response rate of 30% 
(14). Nonetheless, it is likely to suffer from non-response bias. 
We were unable to estimate non-response bias because we 
collected demographic information on callers completing our 
questionnaire and NHS 111 collects demographic information 
on patients. However, we do know that our respondents were 
more likely to report being given advice to use the ambulance 
service than NHS 111 routine statistics reported for the same 
time period (20% in our survey versus 11% in routine data) 
and less likely to report self-care only (10% in our survey ver- 
sus 30% in routine data). Given that respondents with ambu- 
lance referrals found NHS 111 more helpful than those asked 
to self-care, this survey is likely to overestimate the percentage 
of positive views of NHS 111, although in practice this is a 
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small overestimate and remains within levels found in the exist- 
ing literature on other similar services. Two other issues suggest 
that our survey may give a more positive view of acceptability 
than might exist amongst users. First, satisfaction surveys are 
known for reporting high levels of satisfaction, with concerns 
that they may not uncover dissatisfaction, or that low expecta- 
tions may be the cause of high satisfaction (4,5). Second, some 
of the questions we used were positively worded and this may 
have resulted in positive findings. Because of these problems, 
we also included an open question to allow people to express 
positive and negative views and indeed these highlighted prob- 
lems with relevance of questions and advice given. Finally, this 
report of users' acceptability must be considered alongside 
other aspects of the new service (17). The wider evaluation 
found that in its first year of operation in four pilot sites NHS 
111 did not deliver the expected benefits of improving access 
to urgent care or shifting the care of patients to urgent rather 
than emergency care. Indeed, it increased the use of emergency 
ambulance services (17,19). 

Implications 

Satisfaction rates with the pilots of this new service in their 
first year were high and many users wrote positive comments 
about it. This must be interpreted in the light of a ceiling effect 
when measuring patient views and the probability that non- 
response bias resulted in an over estimate of positive views. 
We used open questions to ensure we could identify aspects 
of dissatisfaction and highlighted two areas for improvement 
for the service to address. Some users had a problem with the 
relevance of questions asked. Service providers may argue that 
some questions need to be asked to minimize the risks of miss- 
ing health problems and that a balance is required between 
safe operation of a service and user acceptability. It is likely 
that there will need to be some trade-off between clinical 
requirements for safe practice and user satisfaction. However, 
our view is that the frustration expressed by some users in this 
survey highlights a need to revisit the content of algorithms 
used within the software. An expert panel assessing call record- 
ings, undertaken in our wider evaluation, also identified seem- 
ingly irrelevant questions and suggested that this might affect 
some groups and conditions more than others (17). Some 
of our respondents also expressed concerns about the accu- 
racy and appropriateness of advice given. It may be that the 
most appropriate advice from a clinical viewpoint is not the 
advice people wish to hear. However, again, the expert panel 
in our wider evaluation identified some cases where the advice 
seemed inappropriate (17), suggesting that refinement of some 
algorithm endpoints may be needed. 

The two issues that caused frustration to users may be related 
to the software used or to the use of non-clinical call takers for 



triage. The clinical assessment software is a core ingredient of 
NHS 111 and we would suggest that the first step to improving 
the service would be consideration of adjustment of some of the 
questions and algorithms within the software where particular 
problems have been identified. It may also be the case that a more 
cautious approach to decision-making is required when using non- 
clinical rather than clinical call takers, or they may have less leeway 
than clinicians to override advice recommended by the software, 
although research suggests they operate in a similar way to cUni- 
cal call takers (20). What is interesting is that the open comments 
on our survey did not show positive or negative reaction to non- 
clinical call takers. It may be that people are used to this because 
call takers for the emergency ambulance service are non-clinical, 
that users obtained clinical advice when they needed it or that the 
dissatisfaction with relevance of questions asked within NHS 111 
was partly due to the use of non-clinical call takers. 

There will always be some dissatisfaction with any service 
and it is important to consider whether change to NHS 111 is 
necessary. A core aspect of NHS 111 — the chnical assessment 
software — may not be as accommodating of the diverse range of 
individual situations faced by users as it could be. It is important 
to address this because patients in the current health system in 
the UK have options other than NHS 111 and can select them 
in the face of frustration with the service, e.g. call 999 or attend- 
ance at an emergency department. 
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